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ABSTRACT 


In this paper we present the results derived from a project aimed to study the learning gains obtained by undergrad 
engineering students that used two mobile learning resources (in video format). We conducted our study over a 6-year 
time interval applying pre-test and post-test instruments to both experimental groups (which used the mobile resources) 
and control groups (which did not). Our sample consisted of N = 793 students and we found that the experimental group 
obtained learning gains about 26% higher than those of the control group. We found this difference to be meaningful, 
with p-values around 0.022. Perception questionnaires applied to the control groups showed that most of the students 
consider that the use of mobile resources improved their concept-comprehension and helped them to develop their 
problem-solving skills. 
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1. INTRODUCTION 


The use of mobile technologies has changed the way we interact and communicate (Kukulska-Hulme, 2007; 
Ally and Tsinakos, 2014). Nowadays, it is very common for students to have access to different sources of 
knowledge on the Internet, in many cases in video format. One example may be the so called YouTube 
channels. Using video and multimedia resources in on-line systems is nowadays a must practice and turns out 
to be quite natural for current new-millennial students, from elementary school to undergrad and graduate 
studies. Gamification has also become a major trend in educational institutions and have a strong potential in 
years to come. One of the main advantages of using videos for instructional purposes is that the user can 
watch them anywhere, anytime, as many times as needed and can be paused if necessary in difficult or 
fast-paced parts of the video. 

To the best of our knowledge, however, and despite the importance of using (mobile) videos for academic 
purposes, there has been little effort aimed to set a quasi-experimental environment where the effectiveness 
of using such online material can be assessed (Cochrane, 2014 and Cochrane & Narayan, 2016). On one 
hand, some authors have reported that the use of mobile learning (henceforth mL) resources have a positive 
impact on learning outcomes (Wu, et.al. 2012; Kearney, et.al. 2012; Merayo, et.al., 2015). However, on the 
other hand, some authors have pointed out the potential risk of using mobile devices, since they can easily 
become a distractor in the classroom (Robledo, 2012). 

A decade ago, the Tecnolégico de Monterrey launched a large-scale mobile learning model in which each 
new student was given a mobile device (back then it was a Blackberry) in order to deliberately incorporate 
the use of mobile resources (mainly in video-format) to deliver course content, both in high school and 
undergrad courses. Since then, our research group started an initiative aimed to design, implement and 
evaluate the impact that using mL resources may have in student outcomes. The contribution of this paper is 
then to share with the mL community our findings regarding this project. In Sec. 2 we comment on the 
general aspects of the mL resources design. In Sec. 3 we describe the methodology employed in this study, 
and in Sec. 4 we present our main results regarding the learning gains attained by students whom used the 
mL resources in their Physics courses (the Experimental group) as compared to those obtained by students 
whom did not used the resources (the Control group). We also summarize our findings regarding the general 
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perception of the Experimental students about the use and benefits of using mL resources. Previous results 
have been reported by Robledo-Rella et al. 2010; Chirino et al. 2010; Robledo-Rella et al. 2011). 
Complementary and extended results to this paper are presented by Robledo-Rella et al. 2017. 


2. DESIGN OF ML RESOURCES 


As mention above, we designed several mL resources in video-format to be used in Physics I (Classical 
Mechanics) courses in our institution in order to cover the main themes of the course (from vectors in 3D to 
rotational dynamics). However, for this study we focused ourselves in only two main themes: i) Particle 
Dynamics (Newton’s 2" law and Free Body Diagram) and ii) Conservation of linear momentum. The design 
of these mL resources was carried out by professors of the Science Department with the support of 
instructional designers whom helped to guarantee that the final product videos were adequate and attractive 
to the students regarding i) aesthetic, ii) sound and graphics considerations, and iii) length and content 
displayed. Each video has a framing introduction, a brief description of the main concepts (the theory) and 
ends with a proposed exercise for the student (the practice) so to recap on what was just learned. 

Our first resource was a 5-minute video showing the student how to build a Free Body Diagram (hereafter 
FBD) and how to apply Newton 2" law to a block resting on an incline and being acted by an external force. 
Regarding the theme of Conservation of Linear Momentum (hereafter CLM) we designed two short videos, 
one explaining general aspects of linear momentum in everyday life and the other explaining the conditions 
for linear momentum conservation and the different types of collisions in 1D (this last video included 
measuring experimental cart-collisions in the lab). At the end of the video, the students are also asked to 
answer some summarizing questions. The CLM mL resources were later redesigned so to improve both 
visual and audio elements (Neri et al. 2016). 


3. EXPERIMENTAL METHODOLOGY 


As mentioned above, we implemented a pre-test/post-test methodology applied to Experimental and Control 
groups. The Experimental groups had access to the mL resources for about two weeks using their mobile 
devices (either smartphones or tablets), while the Control group was given similar material in a traditional 
way. In both cases, the professor gave her lectures as usual. In order to minimize the professor-variable, we 
guaranteed that the professor had, for any given semester, at least one Experimental group and one Control 
group. The selection of both the Experimental and Control groups were random before the start of the 
semester. 

Both the FBD and CLM mL resources were applied from 2009-II to 2015-I terms. Table 1 below shows 
the number of students participating in the Experimental and Control groups, both for the FBD and CLM mL 
resources, respectively. We analyzed N = 423 student results for the FBD and N = 370 students for the CLM 
mL resource. Before analyzing our data we cleaned our sample due to misclassified students between the 
Control and Experimental groups (about 2% of the sample). 

The pre-test and post-test were basically the same and were carefully designed so to measure the 
fulfillment of the leaning objectives stated in each mL resource. The pre-test was applied to the whole 
population before the Experimental groups had access to the mL resources. The post-test was also applied to 
the whole population after the Experimental group interacted with the mL resources, about 2-3 weeks after 
the pre-test was applied. In this way we were able to assign to each student a learning gain defined by: 
Gj = Post; — Pre;, where Post; and Pre; are the post-test and pre-test grades of student i. The pre-test and 
post-test were applied to all students during class time in the classroom and each lasted about 15 — 20 
minutes. Each test was graded in a 0 — 100 scale following a well-defined rubric. 

In order to proceed with the analysis, we defined for each group its average learning gain as: <G> = (1/N) 
=xGi, where N is the number of students in each section per semester. 
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4. RESULTS AND DISCUSSION 


Our main results are summarized in Table | both for the FDB and CLM mL resources. We show the number 
of sections/semesters in which the measurements were taken, the number of students, the average pre-test, 
average post-test and average learning gain, both for the Experimental and Control groups. For the sake of 
clarity, we do not indicate the standard deviations of the pre-test and post-test, which were typically of about 
15 points. 


Table 1. Average pre-test, post-test and learning gains for the Experimental and Control groups for the FBD and CLM 
mL resources 


N sections WN students <Pre> <Post> <G> 
FDB Experimental 13 253 34 60 26 
FBD Control 11 170 32 51 19 
CLM Experimental 10 208 34 61 27 
CLM Control 9 162 35 51 16 


As can be seen form Table 1, the average pre-test is similar for the Experimental and Control groups, as 
expected, given that these groups were randomly chosen. 

In order to help us to visualize the differences between the Experimental and Control groups, we show in 
Figure | the “Hake diagram” of the average learning gain <G> vs. the average pre-test <Pre> of each of the 
24 sections considered for the FBD mL resource and the 19 sections considered for the CLM mL resource, 
respectively (Hake, 1988). 
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Figure 1. “Hake diagram” of average learning gain <G> vs. average pre-test <Pre> for each of the studied sections 
for the FBD and CLM mL resources, respectively 


As can be seen from these plots, the Experimental sections tend to populate larger values of <G>, for a 
given pre-test, as compared to the Control sections. 

We performed a t-test to the individual student learning gains (Negp = 423 and Ncim = 370) in order to 
determine if the differences between the Experimental and Control groups were meaningful. For the FBD we 
found p-values of p = 0.0022, making the difference meaningful for this resource. However, for the CLM we 
found p = 0.045, which means that the difference is barely meaningful. 

From Table 1, we can see that, for the FBD mL resource, the average learning gain for the Experimental 
group is about 7-8 points higher (in a 0 — 100 scale) than that for the Control group. For the CLM mL 
resource, this difference is up to 11 points. 
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Although it is not shown in Table 1, we found that, contrary to any reasonable expectation, some students 
within our sample got negative learning gains (that is Post; < Pre;). We found that the percentage of students 
with negative learning gains was about 3% smaller in the Experimental than in the Control groups. 

As mentioned above, we also applied a perception questionnaire (with a 5-step Likert scale) to find out 
what was the overall student perception about the usefulness of the mL resources to enhance 
concept-understanding and the development of problem-solving skills. We collected Np = 203 student 
responses and our main results are summarized in Table 2 below, where we have binned the answer in only 
three levels (Agreement, Neutral y Disagreement). As we can see, the use of mL resources has a positive 
approval among most of the students. 


Table 2. Perception questionnaire main results (Np = 203) 


Agreement Neutral Disagreement 
mL helps to understand concepts 67% 23% 11% 
mL promotes problem-solving skills 61% 23% 16% 


5. CONCLUSIONS 


We presented a self-consistent study spanning 6-years about the use of mobile learning resources and their 
impact in the learning process. In general, we found encouraging (and expected) results indicating that the 
use of video materials favored the learning process of Physics concepts as measured by means of a 
pre-test/post-test instrument, with increased learning gains by as much as 26%, being this a meaningful 
difference. It is worth mentioning that the general teaching methods used by the professors lecturing these 
Physics courses did not change substantially along this 6-year time span, so we consider that the observed 
differences in the learning gains between the Experimental and Control groups are mainly due to the use of 
the mL resources. 

The use of mobile resources in education, in the form of short videos has still a strong acceptance among 
most students and should be considered for teachers when preparing their instructional design. Given the 
current wide acceptance and necessity in our daily lives of the use of smartphones, tablets, the Cloud and 
Web 2.0 services, it is expected that enforcing these channels of communication will continue to be a major 
educational feature. Therefore, we expect that the use of mobile devices related to gamification will certainly 
increase in the forthcoming years. 
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